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Abstract 


We  consider  the  problem  of  service  system  deterioration  and  maintenance 
when  there  are  only  small  statistical  differences  in  the  quality  (or  time 

required)  of  each  produced  item  when  the  system  is  in  the  different  states  of 

deterioration,  but  where  those  marginal  differences  are  economically  important. 
This  is  somewhat  analogous  to  the  situation  in  the  modelling  of  queues  in  heavy 
traffic,  where  the  main  effects  which  are  dealt  with  might  also  be  considered 
to  be  ’marginal’  ones.  In  our  case,  the  production  of  each  item  takes  a  random 
length  of  time  and  the  deterioration  during  any  production  or  sampling  period 
can  have  a  fairly  general  (and  state  dependent)  statistical  relation  with  this 
time  and  with  the  quality  of  the  production.  Due  to  this  generality,  there  are 
several  continuous  parameter  interpolations  (of  the  sequence  of  conditional 
probabilities  of  the  system  states,  given  the  observed  data)  which  are  appropriate 

for  purposes  of  the  weak  convergence,  each  with  its  own  advantages.  [One  can 

work  with  the  ’natural  time  scales’  of  the  deterioration  process,  or  with  that  of 
the  sampling  process,  or  with  something  in  between.]  The  diffusion  process  limit 
is  obtained  when  the  random  sequences  (time,  quality)  are  appropriately 
correlated.  The  limit  process  is  of  the  form  of  a  filtering  problem  for  white 
noise  corrupted  observations  of  a  function  of  a  Markov  chain,  but  the  limit 
problem  is  somewhat  non-standard  since  the  effective  noise  covariance  and  the 
signal  part  of  the  effective  observation  might  depend  on  the  current  conditionals 

probability,  due  to  the  nature  of  the  ’scaling’.  ^ - 

For  one  classical  case,  it  is  shown  that  the  optimal  repair  policy  for  the 
ideal  limit  diffusion  model  is  nearly  optimal  for  the  actual  physical  process  and 
average  cost  per  unit  time  problem),  when  compared  with  the  optimal  policies 
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for  the  physical  process.  The  method  can  be  used  for  all  the  other  cases  also. 
The  result  is  useful  since  the  optimal  policies  for  the  physical  models  are  not 
usually  possible  to  get.  It  illustrates  the  power  and  potential  of  the  diffusion 
approximation  approach  for  such  ’nearly’  optimal  control  purposes. 


Key  Words:  weak  convergence,  diffusion  approximation,  break  down  and  repair 
models,  nearly  optimal  control  policies,  deteriorating  systems  in 
heavy  traffic. 
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Introduction 

In  the  last  decade  or  two,  there  has  been  a  considerable  effort  directed 
towards  getting  (reflected)  diffusion  approximations  of  suitably  scaled  queues  or 
networks  of  queues  in  heavy  traffic:  Lemoine  [1978],  Harrison  [1978],  Reiman 
[1984],  We  are  concerned  with  a  somewhat  non-classical  problem  of  sampling 
inspection  and  repair  which  is  motivated  by  the  same  basic  concerns  that 
underlie  the  heavy  traffic  work.  Let  Qc(t)  denote  the  (Revalued)  content 
(the  number  either  in  service  or  waiting  for  service  at  the  various  servers)  of 
the  system  at  time  t.  In  the  heavy  traffic  work,  loosely  speaking,  the  arrival 
and  service  rates  are  such  that  as  some  parameter  €  goes  to  zero,  so  does  the 
difference  of  the  service  and  arrival  rates.  For  Xf(t)  =  vrQ(t/«),  the  references 
show  that  (Xe(  ■ )}  is  tight  in  the  Skorohod  topology  on  Dr[0,CD),  and  that 
X£(  ■ )  converges  weakly  to  an  appropriate  reflected  Wiener  process  or 
diffusion. 

Typically,  service  rate  minus  arrival  rate  =  0(vr)  at  each  server,  and  the 

drift  in  the  limit  Wiener  process  or  diffusion  is  (loosely  speaking)  proportional 

to  lim  [service  rate  -  arrival  rate].  Apparently,  in  many  applications,  this 
€  /r 

’marginal  drift’  is  economically  important.  For  one  example,  in  the 

manufacture  of  computer  chips,  the  efficiency  is  typically  very  high,  the 
processors  (servers)  are  usually  heavily  loaded,  and  productivity  differences  of 
one  percent  or  so  can  give  a  meaningful  competitive  edge. 

With  the  scaling  used  in  such  problems,  it  is  implicitly  implied  that  the 
value  or  opportunity  loss  for  an  idle  server  is  0(/r)  per  potential  service 
period.  Of  course,  one  primary  motivation  is  that  the  diffusion  limits  are 
usually  much  easier  to  study  or  to  control  than  are  the  actual  physical 
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processes,  and  their  analytical  or  optimal  (if  controlled,  (see  Kushner  and 
Ramachandran  [1987]))  properties  can  shed  considerable  light  on  the  behavior  of 
the  physical  processes. 

Taking  our  motivation  from  the  above  considerations  of  the  heavy  traffic 
problem,  it  is  natural  to  study  the  system  deterioration,  inspection  and  repair 
problem  (the  maintenance  and  repairman  problem  with  sampling  inspection) 
where  (a)  marginal  returns  are  economically  important,  (b)  each  individual 
inspection  or  service  interval  has  relatively  little  effect  on  the  overall  system, 
(c)  the  system  and  statistics  are  sufficiently  complicated  (or  are  non-Markov)  so 
that  the  optimal  repair  policy  is  very  difficult  or  impossible  to  obtain,  (d)  the 
appropriate  diffusion  limit  (for  the  conditional  distribution  process  of  the 
system  state  given  the  observed  data)  and  its  associated  ’repair’  problem  (once 
that  is  identified)  is  relatively  simple  to  solve  or  is  known,  (e)  one  can  show 
that  applying  the  optimal  policy  for  the  limit  process  and  cost  to  the  physical 
system  and  associated  cost  yields  a  cost  (average  cost  per  unit  time  criterion) 
which  is  almost  as  good  as  that  obtained  for  the  actual  optimal  policy  for  the 
physical  system  in  the  sense  that  the  difference  in  the  relevant  costs  converge 
to  zero  as  c  -*  0.  Here,  €  is  a  scaling  parameter  which  ’measures’  the  relative 
differences  between  production  quality  in  the  different  system  states.  The 
’limit’  is  always  in  the  sense  of  weak  convegence  in  the  Skorohod  topology  on 
DIO,*)  (Billingsley  [1968],  Kurtz  [1981]). 

In  our  problem  the  production  or  service  system  deteriorates  in  time, 
samples  of  the  production  are  observed,  and  repair  undertaken  when  called  for. 
As  in  the  usual  repairmen  model,  the  repair  can  be  either  total  or  partial. 
Following  the  usual  spirit  of  the  heavy  traffic  modelling,  where  ’marginal’ 


differences  in  production  are  important,  the  average  percentage  of  ’bad’  items 
produced  would  vary  ’marginally’  with  a  parameter  «.  We  introduce  the 
precise  model  below.  Consider,  for  example,  the  case  where  the  system  is  either 
in  a  good  (G)  or  in  a  bad  (B)  state,  and  that  the  quality  of  the  produced  object 
is  either  good  or  bad.  Owing  to  the  importance  of  marginal  returns,  the  value 
of  a  defective  (bad)  item  might  be  of  the  same  order  as  the  opportunity  loss 
attained  when  the  server  is  idle.  Thus,  loosely  speaking,  P{bad  item  produced 
|  B}  -  P{bad  item  produced  |G}  =  0(vT). 

The  problem,  as  formulated  here,  has  a  number  of  novel  features  (and 
difficulties).  The  form  of  the  limit  filtering  or  estimation  problem  is  not 
obvious  a  priori,  and  the  presence  of  the  control  or  repair  strategy  involves 
questions  of  ’admissibility’  of  the  limit  controls  for  the  limit  problem.  Also,  we 
treat  correlated  observations  or  service  intervals.  Additionally,  the  service 
intervals  are  allowed  to  depend  (statistically)  on  the  unknown  state  of  the 
system.  Also,  since  weak  convergence  on  D^O,*)  tells  us  little  about  the  ’large 
time’  values  of  the  processes,  some  additional  work  is  needed  to  deal  with  the 
average  cost  per  unit  time  problem. 

The  salient  features  of  the  problem  are  illustrated  by  the  single  server 
case,  and  we  stick  to  that  case  until  the  comments  at  the  end  of  the  paper.  Let 
At*  denote  the  time  required  to  process  the  nth  item.  Until  the  end  of  the 
paper,  we  suppose  that  all  items  are  inspected.  This  is  just  for  the  sake  of 
notational  simplicity  in  the  main  part  of  the  development,  and  does  not  affect 
the  final  result.  The  sampling  could  be  at  fixed  or  random  intervals  also. 

For  notational  convenience  and  w.l.o.g.,  assume  that  the  server  is  working 
all  the  time.  If  the  processor  does  not  deteriorate  when  not  in  operation,  then 
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this  assumot'on  is  not  restrictive  at  all. 


On  the  other  hand  if  the  processor 
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does  deteriorate  when  not  in  use  -  but  at  a  different  rate,  or  if  there  are 
several  classes  of  uses,  each  with  a  different  effect  on  the  deterioration,  then 
we  can  let  the  k*  below  depend  on  the  class  of  use.  If  the  sequence  of  the 
classes  of  the  arrivals  can  be  imbedded  in  a  finite  state  Markov  process,  then 
the  development  and  results  are  essentially  the  same  as  what  follows. 

The  System  Model,  Let  S(  )  denote  a  continuous  parameter  Markov  chain 


8 


on  I  =  {1,2,  •  • -,M},  with  infinitesimal  transition  probabilities  {p^}.  The 
value  of  S(  )  denotes  the  deterioration  state  of  the  system,  but  the  rate  at 
which  ’time  progresses’  for  S(-)  depends  on  how  the  deterioration  takes  place 
as  a  function  of  use.  After  the  nth  item  is  produced,  we  observe  {*,  a  bounded 
random  variable,  one  of  whose  components  is  At*,  and  the  other  component(s) 
concern  the  quality  of  the  item.  Let  33*  denote  the  o-algebra  generated  by 
{{J.- and  k(  )  a  bounded  measurable  function.  Define  k*  =  k(£*)  and 

n 

R„  ■  I  k*.  We  could  let  N  -  <■>,  without  much  additional  work,  but  the 

l 

additional  notation  and  detail  probably  outweigh  the  advantages.  S(  )  is  the 
underlying  deterioration  process.  The  ’effective  time  step’  k*  for  S(  )  per 
observation  is  the  deterioration,  and  can  depend  statistically  on  the  ’quality’ 
observation. 

We  use  the  following  system  deterioration  model.  The  V  scaling  is 
consistent  with  the  usual  heavy  traffic  assumptions  -  here,  the  deterioration  is 
slow  on  the  ’real’  time  scale  -  but  goes  at  the  ’normal’  rate  on  the  interpolated 


time  scale.  The  system  state  at  the  completion  of  the  nth  production  is 


defined  to  be  S*  -  S(€R*).  We  use 


v/vw  -rr 
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(1-0  P{S‘  +  1  “  0|Sn  -  «•  Cl>  «  P^+l  =  e!Sn  =  «*•  Cn.!> 

=  «k(«J|+i)PoB  +  0(°- 

One  special  case  is  where  k((*)  =  k0,  a  constant.  Here,  the  deterioration 
process  is  just  a  function  of  the  number  of  items  produced.  If  k((*  +  1)  = 

k0At*  j,  then  the  deterioration  rate  is  a  function  of  the  actual  processing  time. 

The  form  (1.1)  allows  the  deterioration  which  takes  place  during  the  nth 
processing  interval  to  be  correlated  with  the  quality  of  the  item  produced  in 
that  interval.  More  generally,  we  could  let  k(  )  depend  on  a  =  S*  also. 

Define  P«(«)  =  P{S*  =  «|D‘}.  To  get  P*  +  1  from  P«,  <«  +  1,  D«,  we 

need  to  take  the  new  observation  l*+i  as  well  as  the  possibility  of 
deterioration  on  [R*,  R„+1)  into  account. 

We  will  prove  the  weak  convergence  of  a  continuous  parameter 
interpolation  of  (P^(a)}.  It  will  be  shown  that  the  limit  process  is  of  the  form 
of  an  optimal  filter  for  certain  white  noise  corrupted  observations  on  a  jump 
Markov  process  model.  The  exact  form  of  the  limit  process  is  far  from 
obvious,  however.  Ultimately,  our  interest  is  in  the  control  -  repair  problem.  We 
show  that  the  limit  filter  is  the  filter  for  a  continuous  parameter  sampling  and 
repair  problem  for  which  the  optimal  solution  is  known  for  some  important 
special  cases,  and  can  be  computed  as  the  solution  to  a  control  problem  for  a 
diffusion  process  in  others.  Furthermore,  the  optimal  policy  for  the  limit  is 
’nearly  optimal’  for  the  actual  physical  process,  when  compared  to  the  optimal 


policy  for  the  (fixed  and  small  «)  physical  process. 


Continuous  parameter  interpolations.  Owing  to  the  two  ways  of  measuring 
the  passage  of  time  which  are  implicit  in  our  prolem  (the  scale  which  increases 
as  do  the  number  of  observations  n,  and  the  scale  which  increases  as  does  the 
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detcrioration  R*),  there  are  several  continuous  parameter  interpolations  which 
make  sense  for  the  physical  process  each  with  its  own  advantages  from  the 
point  of  view  of  applications  (see  below).  We  now  describe  three  basic 
interpolations.  The  analysis  is  virtually  the  same  for  all  of  them.  Define 

n  n 

R € ( t )  =  €  E  k*  on  [n«,nt  +  «),  T£(t)  =  €  E  At*  on  [n«,n«  +  e), 

1  1 

n  n 

R * 1  ( t )  =  maxftn  :  1 1  k^  i  t},  T"1  (t)  =  max  (n  :  c  E  At*  S  t). 


Interpolation  I.  ( Deterioration  process  time)'.  Define  P^(t)  =  P 


R'At)/ , 


(i.c.,  it  equals  P*  on  the  interval  [R*,R*+1),  anc*  sct  S*(t)  =  S(t).  Here  the 
progress  of  time  is  marked  by  the  rate  of  system  deterioration,  and  Rj^tJ/e 
is  the  number  of  items  produced  by  interpolated  time  t. 

Interpolation  2.  (Real  time).  Define 

P«(t)  =  P*  on  (T*,T*+1)  and  S*(t)  =  S^fT^t))). 

Our  S€ ( t )  here  is  the  actual  state  of  the  system  when  [t/e]  samples  are 
taken. 

Interpolation  3.  ( Sampling  time).  Define  P^(t)  =  P*(ot)  on  [ne,ne  +  €), 
S*(t)  =  S(R*(t)).  This  is  the  ’usual’  interpolation  for  (P*(a)},  since  each  ’step’ 
or  sample’  corresponds  to  «  in  the  interpolated  time  scale. 

We  work  with  the  interpolation  1  -  in  the  deterioration  or  system  state 
time.  Later,  (Section  4)  we  (trivially)  convert  the  result  for  that  case  to  the 
result  for  interpolation  3.  The  other  case  is  similar.  Interpolation  1  perhaps  is 
most  natural  for  the  pure  filtering  problem  -  since  the  time  scale  is  that  of  the 
signal  S(  )  (hence  the  evolution  operator  involved  in  the  drift  term  of  the  limit 
process  is  that  of  S(  )),  even  though  the  observations  (samples)  are  taken  at 
random  intervals.  The  interpolation  3  seems  to  be  most  natural  for  the  control 
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problem  in  the  sense  that  (loosely  speaking)  the  mean  costs  associated  with  'bad 
production’  are  proportional  to  the  (interpolated)  time  that  the  system  is  in  a 
’bad’  state.  But,  the  frequency  of  repairs  (and  the  associated  repair  costs)  is 

most  meaningful  in  interpolation  2.  In  fact,  simple  rescalings  take  us  from  one 
case  to  another. 

Section  2  contains  the  basic  assumptions  and  the  definition  of  the  terms  in 

which  the  limit  filter  will  be  defined.  Section  3  deals  with  the  weak 

convergence  of  the  true  physical  filters  to  the  limit  filter.  Various  extensions 

are  dealt  with  in  Section  4:  intermittent  sampling,  other  interpolations  and 
’non-anticipative’  properties  of  limits  of  certain  functionals.  In  Section  5,  we 
show  that  the  limit  filter  can  be  used  to  get  a  nearly  optimal  repair  and 
maintenance  policy  for  the  actual  physical  system  for  a  cost  function  of  the 
average  cost  per  unit  time  type. 


2.  Assumptions  and  Definitions 

For  motivation  of  our  definition  of  the  observation  model  {(£)  given 

below,  first  consider  the  classical  case  where  the  state  space  is  (G,B).  Let  U£) 
be  i.i.d.  under  each  G  and  B  (the  most  common  asumption  in  inspection  ■ 
repair  models).  The  discussion  in  the  last  section  on  ’marginal’  differences  can 
be  quantified  by  writing,  for  example, 

(2.1)  P£(0  -  P{{*  e  +  dO|S«  -  G)  -  [PB(0  +  /r  0U)]X(dO. 

where  p£,  P|  are  densities  with  respect  to  the  given  measure  X(  ).  [We 

could  replace  B(  )  by  Bf()  where  0£(  )/B(  )  1.]  Let  B(0/Pb(0  be 

bounded  and  J*0(OMd{)  =  0  (required  for  the  probability  to  integrate  to 

unity).  If  /rB()  is  replaced  by  a  term  of  smaller  order,  then  as  £  -  0,  B 
and  G  are  asymptotically  indistinguishable  by  the  observations.  If  the  order 

is  greater  than  yr,  then  as  £  -  0,  one  can  distinguish  B  and  G  in 

essentially  ’zero  interpolated  time’. 

If  the  {$£}  are  correlated,  then  a  more  complex  model  is  needed.  The 
next  most  complex  model  involves  a  first  order  Markov  dependence,  where  we 


use  (a  =  G  or  B  here) 


pu*+l  €  A|<£  =  S£  -  a) 


jp^.r 


)X(d(') 


(2.3)  P£(U')  “  PB(U')  +  /rSU,{').  JflU.DW)  -  0, 

where  PG({,  ),PBU,  •)  are  densities  relative  to  a  measure  X(  )  for  each  (, 
and  B({,{  ')/PBU,{  ')  is  bounded. 

We  prefer  to  work  with  the  more  complex  Markov  dependence  model  in 
this  paper,  since  it  illustrates  the  possibile  directions  of  further  generalizations 


and  allows  (state  dependent)  correlation  in  the  service  time  or  ’quality’  sequence 
-  even  though  the  notation  is  more  complicated  than  that  for  the  i.i.d.  case. 

For  the  state  space  I  =  {1,2,-  - -,M}  case,  we  extend  (2.2),  (2.3)  as 
follows,  where  P0((,-),  P^t, •)  are  densities  with  respect  to  a  measure  X(): 
There  is  P0: 

(2.4)  Pf(U’)  =  Pq(U')  +  ✓r0iU,?'). 

[Again  0£  :  0£/0  -*  1  could  be  used.]  Although  our  observations  (£  are 
finite  dimensional,  it  is  not  necessarily  true  that  the  dimension  of  the 
observational  process  which  appears  in  the  limit  filtering  problem  is  finite 
dimensional  (a  surprise!).  It  will  be  finite  dimensional  only  if  (2.4)  takes  the 
special  factored  form 

N 

(2.5)  Pf(U’)  =  P0(U')  +  vr  £  B/OP/U'), 

i=i 

for  some  N  <  ®.  We  will  use  this  factored  form.  It  seems  that  any  problem 
can  be  approximated  arbitrarily  well  by  such  a  form. 

Assumptions. 

A2.I.  In  (2.4),  (2.5),  the  P;U,* '  )/P0U,t ' )  are  bounded  uniformly  in 
and  |Pj((,(  ')X(d( ')  =  0,  all  i,(.  The  range  of  (  is  compact. 

A2.2.  Define  the  one-step  transition  function  P((,l,d(')  =  P0((,( ')X(d( '), 
and  let  the  associated  Markov  chain  have  a  unique  invariant  measure  M°(  )- 

We  use  E°  to  denote  the  expectation  with  respect  to  p°{  ).  The 

assumption  seems  unrestricted  and  trivially  holds  in  the  i.i.d.  case,  where  we 


have  P0U,t')  =  P 0U‘),  etc.  and  jt°(dO  =  P0(d$)Md5)-  Note  that  the  ’noise’ 
which  appears  is  not  that  of  the  observations  directly,  but  rather  how  the 
observations  ’affect’  the  marginal  differences  in  the  distributions. 


A2.3.  The  matrix 


£,  -  fE.p<» u  <  N) 

is  nonsingular.  (This  is  rather  unrestrictive  -  if  it  were  singular,  we  would  use  a 
representation  (2.5)  of  lower  dimension.  In  the  i.i.d.  case,  the  E°  is  redundant.) 


A2.4.  k(-)  and  At6  are  bounded,  measurable  and  inf  k($)  >  0,  inf 

At€  >  0.  There  are  measurable  functions  k(),  At()  such  that 

Sj.1-  i.  ^n-l  =  «  =  EW^)pa-l’Sn-l-  *1  »  W.O. 


EK\*ll  '  *n-l  -  l]  = 


anrf  k  (i,  •)  and  At(i,()  are  continuous  for  each  i. 


A2.5.  Jk(s,l')P0U,l')\(dl'),  Jm(s,*')P0(U')W), 


a(u2  PiC^t1) 
p0(«,n 


x(cU') 


are  continuous  in  (,  for  each  s. 


For  any  function  h(  )  on  I  and  P  *  {P;},  define  h(P)  =  E  h(i)Pi. 

i 

The  conditions  on  At  in  (A2.4,  A2.5)  are  needed  only  for  interpolation  2. 


The  general  filtering  equation  for  a  Markov  chain  signal  and  white  noise 
corrupted  observations.  For  future  reference  and  for  purposes  of  identifying 
the  weak  limit  of  {P£(  ),S€( •)),  we  write  the  general  equation  for  the  evolution 
of  the  conditional  probability  distribution  for  this  classical  filtering  problem. 
Let  H?(  ),  i  <  M,  be  bounded  continuous  real  valued  functions  and  w?(-), 
i  <  N,  stationary  Wiener  processes,  independent  of  S(  ),  and  with  correlation 
matrix  E0  =  <Ew°(l)w°(l),  i,j  <  N}.  Write  H°  =  (H°..,H°)\  y°(t)  = 

JoH?(S(T))dT  +  w9(t),  y°  =  (y$,- • -.y^)'.  Define  the  conditional  probabilities  Pg(t) 
=  P{S(t)  =  s|y°(T),  T  <  t}.  Write  P(t)  =  {P;(t),  i  <  M}.  Then  (Liptser  and 
Shiryaev  [1977],  Elliot  [1982]) 

(2-6)  dP,(t)  «  [  E  pto  P.  (t)  -  I  Pfi  P.(t)]dt 

i*»  i#* 

+  [H°(s)  -  H°(P(t))]'  E^fdy0  -  H°(P(t))dt]. 


We  use  the  following  definitions  in  the  next  section. 

Define  K(s)  =  Jk(s,On°(dO  =  E°k(s,0.  Define 

Jj=l  P0(U’) 

and  Hj(s)  =  E°hi(s,5)-  Then  H(s)  *  Ej0(s)  *=  (H;(s),  i  <  N),  where  wc  write 
B(s)  -  (B.(s),  •  ■  M))! 


3.  Weak  Convergence  of  (P£()}. 

Theorem  3.1  shows  that  the  (P*(),  i  <  M}  converge  weakly  to  a  filter 
of  the  type  (2.6).  The  form  of  the  filter  is  somewhat  non-standard,  and  is 

discussed  after  Theorem  3.1.  Define  bj(P)  =  E  p^P;,  b2(P)  =  E  ptjP,,  f°r  p  = 

it, 

{Pj,i  <  M),  and  set  b4(P)  -  bJ(P)  -  b,2(P),  b(P)  =  (b,(P),  s  €  I}.  Recall  that 
interpolation  1  is  used  in  Theorem  3.1.  If  k(t*)  =  At*,  then  interpolations  1 
and  2  are  the  same. 


Theorem  3.1.  Assume  (1.1)  and  (A2.1)  to  (A2.5).  There  is  a  Wiener  process 
w(  • )  =  (Wj(  ),  •  •,wN(  ))',  with  covariance  matrix  Ejt  and  independent  of  S(  ) 
such  that  (P*(),  S(  - ),  s  <  M)  ^  (P,(  ),  S(  ),  s  <  M)  in  the  Skorohod  topology  on 
DM+1(0,®),  where 

(3.1)  dP,(t)  -  b,(P(t))dt 

♦  p.(.)  -  atnuna — r  rdy .  wsaui 

Lk(P(0)  K(P(t))J  ^K(P(DV  *-  K(P(t))  -I 

and 


(3.2) 


H(S(T))  .  f‘  dw(T) 
- dT  +  ’ 

o  K(P(T))  * o  K^(P(T)) 


Remark.  Comparing  (3.1)  with  (2.6  )  yields  that  (3.1)  is  a  filtering  equation  for 
the  effective  observation  process 


(3.3) 


dy  «  dy° 


d  w 


K^P(t)) 


H(S(t)) 

+  - dt 

K(P(t)) 


h  dw°  +  H°(S(t),  P(t))dt. 

This  is  somewhat  non-standard  since  the  noise  and  signal  parts  of  the 
observation  process  depend  on  the  estimates  (Pj(t)).  Since  these  are  just 


functions  of  the  past  observed  data,  a  straightforward  extension  of  the  proof  of 
(2.6)  in  Liptser  and  Shiryaev  [1977]  or  Elliot  [1982]  yields  that  the  solution  of 
(3.1)  satisfies  P^t)  =  P{S(t)  *  s|y(T),T  <  t).  The  ’weighing’  of  the  observation 
term  by  K_1(P(t))  arises  owing  to  the  fact  that  the  ’natural’  scale  for  the 
observation  process  is  our  interpolation  3,  the  sample  time  scale,  but  the 

interpolation  which  is  used  to  get  (3.1)  is  our  interpolation  1,  the  deterioration 
process  time.  The  drift  term  in  (3.1)  is  not  ’weighed’,  since  it  is  just  the 
operator  of  the  deterioration  process  S()  in  its  ’own’  time  scale.  The  term 

ra+8a  - 

j  Ul/K  (P(t))  is  (loosely  speaking)  the  average  number  (conditioned  on  the 
’data’  up  to  t)  of  samples  taken  in  the  time  that  S(  )  evolves  from  S(a)  to 
S(a  +  5a).  See  the  proof  below.  In  the  scale  of  interpolation  2,  both  the  drift 
and  diffusion  terms  would  be  scaled  by  functions  of  P(),  unless  k({)  =  At. 
With  interpolation  3,  the  drift  only  would  be  scaled  by  such  a  function,  (sec 
Section  4),  since  there  the  evolution  of  S£(  )  depends  on  the  evolution  of 

Rc(). 

It  is  necessary  that  we  treat  [P£(  ),  a  €  I)  and  S(  )  together,  in  order 
that  the  limit  process  be  interpcrtable  as  a  filtering  process  with  observation 
process  (3.2)  or  (3.3).  Note  that  the  dimension  of  the  observation  process  in  N 

and  the  underlying  signal  S(  )  appears  only  via  H(  ).  Since  the  limit 

conditional  probability  process  (3.1)  (or  the  analogs  for  the  other  interpolations) 
is  much  simpler  than  the  evolution  equation  of  Bayes’  rule  for  our  semi  Markov 
process  model,  it  has  the  usual  advantage  of  diffusion  approximations.  It  can 
be  used  to  evaluate  the  effects  of  different  deterioration  or  sampling  or  repair 
processes.  Furthermore  (Section  5)  it  is  the  correct  process  to  use  to  get  the 
’approximately’  optimal  repair  and  maintenance  policy. 


Proof.  1°.  The  proof  starts  with  Bayes’  rule  and  via  successive 

rearrangements  of  the  terms,  puts  the  expression  in  a  form  where  essentially 
standard  weak  convergence  methods  yield  both  tightness  and  the  desired 
expressions  for  the  limit,  as  well  as  the  independence  of  S(  )  and  w(  )  and 
the  non-anticipativeness  of  the  {P,(  ),s  <  M)  with  respect  to  w(  ■). 

2°.  By  Bayes’  rule,  (where  the  P{ ?„  |  • }  are  the  obvious  densities  with 


respect 

to  X()) 

-  »| 

K+i) 

*  f  p(S*+1  * 

i 

ii°t«»ii-p(s;«  -  siss  -  o  Cii 

(3.4) 

£  P{«5+1 

i 

I8n  * 

».  D‘)  p(s‘  - 

ipJ)pis;«  *  sis;  - 1.  «:«> 

*  r  p<Ct 

!Sn  * 

i.  D^p(S‘  * 

i|®2> 

IPUS+.ISS  -  s,  Dj)P‘(s)<l  -  <  *  0(0) 

"  ♦  £  -  i.  WiX'C.p,.  +  0(0)1 

i* 

*  «  * »  »n>- 

i 

Now,  use  (2.5)  in  (3.4)  in  the  form 

p(C.lsl  -  ’ ■  ■ 

i=i 

Expand  the  resulting  expression  in  powers  of  vt  and  divide  all  terms  by 

-  po(CCi>  Thcn 


I 


$ 

N1 

$ 


Dsr'ne  p"  *  Pjiii'is+i)  a,i[i  ps 


p- ^  RJI  Fim  vn^r-  yir  W  k-v  yy vwv'.  ^  vT^my.v.v.vffywwT>'  ^wv  w-v.^wy.TWin»7W 
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(3.5) 


c 

n+1 


+  VT 


-  € 


P‘+1(s)  -  P*(s)  +  o(€)  -  €bjp«)k„f+l  +  «W< 

2  6/s)  £p*(s)  -  /r  [  EP«(i)  E  B/i)^  ]p‘(s) 

j=l  r0  L  i  j=l  0  J 

[rpjii)  r  e/o  $-][  i  b/s)  ^-]p„'(s) 

L  i  j=l  r0  J  L  j=l  r0  J 

r  N  p"i 1 

♦  <[  I  PJO  £  »/»  ji]  pJ») 

S  P«(s)  +  o(0  +  €  [b*(P«)  -  bJP*)]k*+1  +  Tf  n  +  •  •  •  +  T*n, 


where  the  T*  n  are  the  last  four  terms  on  the  right  of  (3.5).  Rearranging 
(3.5)  and  using  the  definitions  of  b§(P)  and  0/P)  yields  (where  we  write 


K  *  (pn(«)’  «  « 


(3.6) 


Pn+l(5)  «  P^(S)  +  0(0  ♦  €k‘+1b,(P‘) 


+  ^  pfr)  2  (6/s)  *  6/P*)) 

j=ir0 


-  € 


PS*)  <  £  B/Pj  4  >  [  £  ft  <»/*>  ■ 

j=l  0  Lj=lrO  J 


3°.  To  arrange  the  proper  centering  of  the  terms  in  (3.6),  we  next 
evaluate  the  conditional  expectation  (given  D*)  of  the  last  two  terms  in  (3.6), 
which  we  call  /r  P*(s)T ,(s,?;;,{*+1,P*)  and  -€  P«(s)t1(s,?‘,«‘+1.P‘),  resp.  For 
notational  simplicity,  we  sometimes  use  subscripts  to  denote  the  data  being 
conditioned  on. 

We  have,  via  the  definition  h/P,{)  -  E  h/i.DPj  and  (A2.1), 


I 

I 


1;*| 

| 

!«3 


=  «  r 


E  F7?rrr[po^^)  +  ^  E  Pk(^.08^i)])P^(i)Mdo 

i  J  r0'4n’^  k=l 

f  dH€  l)  N 

1  rr^Ti 1  hiuwkwwu  - 

i  J  r0''n’4'  k=X 


From  this,  wc  get 


ED€  "  K^Tl^n+vK) 

n 

(3.7)  N 

-  <Pfc)  I  [8j(S)  -  8/P*)]h/p;,!‘). 
j=l 

By  a  similar  calculation  we  have  (modulo  o(«))  that  the  expression  in  (3.7) 
equals  €P«(s)  E  t2(s,<«,?«+1,P«). 

n 

Thus,  we  can  center  both  terms  of  the  Therms  about  their  conditional 
expectations  and  change  (3.6)  by  only  o(c).  Note  that 

(3.8)  ,p'(S)  [t,<u«  q+rr«>  •  eb<  q+l.ps]  = 

n 

is  a  martingale  difference  term  with  variance  bounded  by  tJK  for  some 

n 

constant  K.  Consequently,  the  continuous  parameter  interpolation  (with  either 

interpolation  1,2  or  3)  of  I  q.€  will  converge  weakly  to  the  zero  process  as 

i 

€  •*  0.  Equivalently,  the  continuous  parameter  interpolation  of  the  sums  of  the 
centered  last  term  in  (3.6)  converges  to  the  zero  process.  W.l.o.g.,  we  henceforth 

,  .  A 

include  this  term  in  the  o(«),  but  center  the  Tj  term  about  its  conditional 
expectation. 

Now,  centering  and  rewriting  (3.6)  yields  (recall  the  definitions  in  the  last 
paragraph  of  Section  2) 


t  »,*  ti*  I*'  to*  •*'  to*  fa*. fa* 


where 


pS+i<s)  ■  p^>  *  »«)  *  <k'+1b,(PS) 

♦  Pj<s)  E  [Bj(s)  -  fyp')]  [Vr  -  <h1(P',!*)] 

i=l  r0 

"  P*(s)  +  0(0  +  €k‘+1b.(P*) 

+  Pn<S>  [H(S)  *  ^K)]^lbYn  '  £h(P‘,0] 

*  P‘(S)  +  0(C)  +  <k‘+1b,(P<)  +  P‘(S)[H(S)  -  H(P«)]  'z^BV' 


8Y«  -  ( ✓F  p«/P0n,  i  <  N),  h(P,0  =  {hi(P,0,  i  <  N). 


and  8V*  is  defined  in  the  obvious  way.  By  the  definition  of  h(P,0,  and 
the  results  above  (3.7)  the  term  in  the  right  hand  bracket  of  (3.9)  (namely 
{6V*},  the  ’approximate  innovations’  sequence)  is  a  martingale  difference  with 
respect  to  the  a-algebra  {D*}  Eqn.  (3.9)  is  now  ’close’  to  the  desired  limit  (3.1), 
and  we  are  ready  for  the  weak  convergence  argument. 

4°.  Tightness.  The  sums  of  the  b^  terms  on  the  r.h.s.  of  (3.5)  interpolate 
to  (interpolation  1),  resp.. 


(3.10a) 


(3.10b) 


-  f 

Jo 

f  bJ(P€( 

*  n 


;(T))dT  +  0(C), 


(T))dT  +  0(c). 


The  processes  given  by  the  integrals  in  (3.10)  are  tight,  their  limits  are 
absolutely  continuous  with  uniformly  bounded  derivatives.  The  last  term  on 
right  (to  be  denoted  by  q*)  of  (3.9)  is  a  martingale  difference  with 
(conditional  on  E*)  variance  bounded  by  cK,  for  some  K  <  ®.  Consequently, 

n 

a  - 

the  continuous  parameter  interpolation  of  E  q*  is  tight  in  D[0,“>)  also  (with 


$ 

!tsl 


$ 

;* 

■:$ 

•'»! 

A 


*>i| 


1 


I 

& 

iTC 

I-#* 

8 

& 


iS 

I 

tt 


S 


m 


any  interpolation  1,2  or  3). 

Henceforth  with  a  slight  abuse  of  notation,  let  «  index  a  weakly 
convergent  subsequence  of  (Pfi  ),  S(),  R£(),  R£l(  ),  i  <  M},  with  limit  denoted 
by  {Pj(  - ),  S(  -),  R(  ),  R_1(  ),  i  <  M).  (It  will  turn  out  that  the  limit  does  not 
depend  on  the  subsequence.)  Thus,  the  terms  of  (3.10)  converge  weakly  to 


-f  bj  (P(T))dT,  [  bi\P(T))dr,  resp. 

J  n  »  n 


Note  that  E  £  eBY«  -  «h(S«,^)  and  define  6w«  *  [BY*  -  ch(S*,«*)J 


and  r<  =  eP«(s)[H(s)  -  H(P*)] 'EjWS*,?')  -  H(S*))  -  (h(P*,t«)  -  H(P‘))].  We 
can  rewrite  the  last  term  on  the  right  of  (3.9)  as 


(3.11) 


P‘[H(s)  -  H(P«)]'  Z\l[vr  6w*  +  £H(S*)  -  €H(P«)] 


+  r*  =  8v*  +  r* 


It  will  turn  out  that  the  continuous  parameter  interpolation  of  I  r* 

i=i 

converges  to  the  zero  process.  The  rearrangement  in  (3.11)  was  done  in  order  to 
extract  what  will  be  the  signal  component  of  the  observation  (namely  H(S*)) 
and  to  replace  the  centering  h(P„'0  *ts  conditional  mean  H(P*).  The 

l*  is  averaged  out  via  the  invariant  measure  jr°(  ).  H(P*)  is  the  conditional 

expectation  of  H(S*)  given  D*,  and  the  resulting  form  (modulo  the  ’error’ 
rn)  is  closer  to  that  of  the  limit  process. 

The  term  /r  P*  [H(s)  -  H(P*)] 'ej^w*  =  q*  is  a  martingale  difference 
(with  respect  to  the  o-algebras  measuring  {(*,  i  <  n,  S*)),  with  third  moment 
being  0(€S^1).  Thus,  the  weak  limits  of  the  continuous  parameter  interpolation 

n 

of  E  qf  (with  interpolation  intervals  being  either  «  or  {tk*})  are  continous 

t 

processes.  Thus,  the  limits  P4(  • )  of  P * ( • )  will  be  continuous. 

5°.  Weak  Convergence.  It  will  sometimes  be  convenient  to  use  the 


raawnwwTWTWTOPBDWiw "jfj*  vv  v  v ,v  \y  v>  v  \« \v  •■>■■»>- ■:■  w>- ,jr»jr»_vMr». r» ytv* v*.-^ -•'  .->■  '.-*• . 
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constant  interpolation  intervals  e  rather  than  (£k£),  obtain  the  limit  and  then 
rescale.  We  will  state  what  the  interpolation  interval  is,  whenever  it  is 
important.  Owing  to  the  random  number  of  samples  taken  on  any  (interpolated) 
time  interval  [0,t],  we  proceed  indirectly.  Define  P;(  • )  =  Pj(R_1(-))  and  S(  ) 
=  S(R’l(-))>  The  limits  R()  and  R'*()  are  continuous  and  strictly 

monotonically  increasing,  and  the  limit  of  RjV)  is  just  R_1( •),  the  inverse 
of  the  limit  of  R£(),  so  there  is  no  ambiguity  in  the  notation.  Define 
S£(  )  =  S(R*tV))  and  Pf(  )  =  Pf(R£1(  ))-  We  work  with  S  and  P  and 
then  rescale.  We  have  (§(•)»  Pj(-),  i  <  M)  *  (S(  ),  P.(-),  i  <  M). 

By  the  definition  of  k(  -), 


Since  the  first  sum  on  the  right  is  a  martingale  with  quadratic  variation 
<  K«2n  for  some  K  <  ®,  the  desired  weak  limit  R(  )  of  [R£()}  is  that  of 
the  continuous  parameter  interpolation  (interval  e)  of  the  last  sum  on  the  right. 

The  averaging  method  which  will  be  used  to  get  the  limit  process  is  the 
’invariant  measure’  method  used  in  Kushner  and  Shwartz  [1984]  for  processes 
with  limits  satisfying  ordinary  differentia]  equations  and  in  Kushner  [1984, 
Chapter  5]  for  limits  which  are  general  diffusion  processes.  However,  the 

development  here  is  self-contained. 

Let  A  >  0  such  that  t  is  an  integral  multiple  of  A  and  let  T  <  t, 

where  t  and  T  are  integral  multiples  of  A.  (The  integrality  is  for 

notational  convenience  only;  it  is  actually  only  required  that  the  ratios  go  to 
infinity  as  t  -*  0  or  A  -•  0.)  Write  De(iA)  =  0£^/£-i.  For  use  below,  we 

evaluate  the  weak  limit  of 


(3.12) 


€  E  ,  E  k(P‘,l«). 
C£(T)n£=T 


Owing  to  the  continuity  of  k(i,)  and  the  fact  that  {P£(  ■)}  is  tight  and  has 
continuous  limits,  the  weak  limit  of  (3.12)  is  the  same  as  the  weak  limit  of  (as 


€  -•  0,  then  A  -*  0) 


(3.13) 


E  E  AE  iA), 

D£(T).iA=t  D'CiA)  La  J 


where  m.  =  i  A/  € . 


For  each  u,  define  the  measure  vf’^(u,  )  by 


v^fw,  )  =  y  ‘I:1  P{J£  €  •  | B€(iA)}  (w). 


The  inner  sum  of  (3.13)  equals 
(3.14)  jk(P£(iA),$)v£-*  (W,d0. 

The  set  of  measures  (v.€'A(u,  ■  ),  u,i,A,€)  is  tight,  since  the  range  of  ^  is 
compact.  Fix  w.  With  an  obvious  abuse  of  notation,  let  t  index  a  weakly 
convergent  subsequence  (of  the  originally  selected  weakly  convergent  subsequence 
of  € )  such  that  v^fw.)  =»  v^(u,  ),  for  some  set  of  measures  vA(u,.),  and  all 
i  <  t/A.  It  will  turn  out  that  this  limit  measure  does  not  depend  on  u,A  or 
i.  We  next  characterize  v^(u,  ). 

By  (2.5)  and  the  definition  of  vtiA(  ),  the  weak  limit  of  (3.15)  (as  €  -  0) 
is  the  same  as  the  weak  limit  of  (3.14),  mod  terms  of  value  o(A)/A  (recall  that 
w  is  fixed  here). 


iW»T».nTiTl] 


111 

*l! 

‘i 

** 

*« 


V 

V 

y 

y 

y 

y 

ti 


(3.15) 


€  mi+i"l  f—  ~ 

7  t  Jk(P«(iA).t;)P!dJ;;| ('...S'.jJ  -  P{d d S*_, |De(ii)} 

n=m-  J 

(  f _ 

=  “  E  I k(P£(iA),  OPoC^.i-OMdO  ■  P{d^.x|D£(iA))  +  o(A)/A 

n  —  m  * 


V  \ 


k(P£(iA),  «P0({'.i,«X(<U)  •  v£-A  +  o(A)/A. 


Using  the  first  continuity  condition  in  (A2.5)  and  taking  limits  in  both 
(3.14)  and  in  the  last  term  of  (3.15)  yields  that 

(3.16)  Jk(P(iA),  5)v(WU)  =  |k(P(iA),t,)P0({,?,)x(d{')vf(w,d{). 

By  letting  k(-,)  be  replaced  by  an  arbitrary  bounded  and  continuous 
function  of  f.  in  (3.15)  and  (3.16)  yields  that  vA(u,  •)  is  an  invariant  measure 
for  the  Markov  chain  with  one  step  transition  density  P0U,S ' ) X{d ^  ').  By  (A2.2), 
this  invariant  measure  u°(  )  is  unique.  Hence  vA(u,  •)  =  ■),  and  does  not 

depend  on  A,u  or  i.  Thus,  the  weak  limit  of  (3.10)  is 


[  E°k(P(u),  «du  =  f  K(P(u))du, 

*  T  '  T 

where  (recall)  E°qU)  =  Jq($)M°(dO  for  any  function  q(  ). 

Let  g()  be  bounded,  real  valued  and  continuous.  By  the  above 
arguments  of  this  section,  for  T;  <  T,  i  <  r, 

lim  E  g(P£(Ti),  i  <  r)[Re(t)  -  R£(T)  -  [  K(Pe(u))du]  =  0. 

'  JT 

Owing  to  the  arbitrariness  of  g(  ),  Tj,r,  t,T,  we  have  that  R(t)  -  JgK(P(u))du  is 
a  martingale  with  zero  quadratic  variation.  Hence  it  is  constant  and  we  have 


W. 
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au 


R(t)  =  [f,K(P(u))du.  It  follows  that  R_1(t)  =  K._1(P(t));  hence  R_1(t)  =  , 

*  0  K(P(u)) 

An  argument  very  similar  to  the  above  ’averaging’  argument  but  using  the 
last  continuity  condition  in  (A2.5),  shows  that  the  interpolation  (interval  «) 

n 

of  €  E  r*  (see  above  (3.11)  for  the  definition  of  r*)  converges  weakly  to  the 
1 

zero  process.  The  limits  of  the  interpolations  (interval  «)  of  the  sums  in  (3.11) 
which  do  not  involve  8w€  are  also  obvious.  I.e., 

n 

*/«  i 

(3.17)  €  E  P‘(s)  H(P‘)  Ej1H(S*) 


converges  weakly  to 
(3.18) 


pfc 

P,R-1(u)H(P(R-1(u)))  'E-1  H(S(R‘1(u)))du. 

»  n 


It  is  not  difficult  to  show  that  there  is  a  Wiener  process  W(  ),  with 

covariance  Ejt  such  that  S(  )  and  P(  )  are  non-anticipative  with  respect 

t/e 

to  W(  • )  and  that  the  process  (see  above  (3.11)  for  the  definition),  E  6vjj 

t 

weakly  to 


pt 

(3.19)  P,(u)[H(s)  -  H(P(u))]  E;1  [dW(u)  +  H(S(u))du  -  H(P(u))du] 

J0 


=  dv(u). 

To  get  the  proper  limits  with  the  original  interpolation  intervals  {^n}’  wc 
actually  need  the  limit  of 

K\t  )/€ 

*  <■ 

n=l 

rR'1^) 

But  this  is  dv(u)  =  vfR'Vt)) 

Changing  scale  yields  the  existence  of  a  Wiener  process  w(  ),  with 


covariance  Ett  such  that  S(  )  and  P(  )  are  non-anticipative  with  respect 


to  w(  )  and 


(3.20) 


v<R-ct»  =  [‘P.(u)  '  r_L__  r1 

J0  •  LK-(P(u))  K(P(u))  J  lK(P(u))J 


dw(t)  H(S(u)) 


dw(t' 

-K.typ(u 


H(P(u)) 


'(P(u))  K(P(u))  K(P(u)) 


We  omit  the  details  concerning  the  mutual  independence  of  S(  )  and 
w(  •).  It  is  basically  a  consequence  of  the  fact  that  as  e  -»  0,  the  distribution 
of  depends  on  S*  only  via  the  vr  term  in  (2.5).  The  proof  of  the 

independence  is  accomplished  by  calculating  the  operator  of  the  join-t  process 
(S(  ■  ),w(  ■ )). 

For  the  ’non-stochastic  integral*  terms  such  as  (3.17),  which  arise  from 
(3.11),  we  need  the  limit  with  t  replaced  by  Rj*(t)  (for  interpolation  1). 
Thus,  the  desired  limit  is  (3.19)  with  t  replaced  by  R‘x(t).  But  this  is  just 
(3.18)  with  R'x(u)  replaced  by  u  and  with  the  integrand  divided  by  K(P(u)), 
as  above.  Adding  these  terms  to  (3.20)  and  (3.10)  (with  £  =  0)  yields  (3.1). 

Finally  we  note  that,  given  S(  )  and  any  w(  )  independent  of  S(  ), 

the  solution  to  (1.1)  is  unique  since  E  Pj(s)  =  1  and  the  coefficients  are 

> 

uniformly  Lipschitz  continuous  in  {Pj,  i  <  n).  Thus  the  chosen  weakly 
convergent  subsequences  are  irrelevent,  and  the  weak  convergence  holds  as  stated 
in  the  theorem. 
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4.  Extensions 

4.1.  Skipping  samples.  Instead  of  sampling  each  item,  let  us  sample  only 
items  B£,  Bf,-  •  •,  where  B£  t  »  and  6£+1  -  B£  is  bounded.  Then 


*2+1  £  . 
€  L  k* = 

i=B:+i 


replaces  the  k£+1  in  the  work  of  the  last  two  sections.  Let  now  denote 

the  observation  which  is  taken  at  the  nkh  actual  sample,  and  let  it  include  k£ 
as  one  component.  Consider  two  particular  cases. 

Case  1:  k({)  =  At.  Under  the  assumptions  of  Theorem  1  with  lc£ 

replacing  k£,  the  conclusions  of  the  theorem  hold. 

Case  2:  Let  k(5)  =  k0,  a  constant.  Then  with  kjB^j  -  B£]  =  Ic£+1 
replacing  the  k£  in  Theorem  1,  the  conclusions  of  that  theorem  still  hold. 


4.2.  A  Tandem  Network.  Consider  a  simple  tandem  system  with  two 
processors  in  series  and  let  S(  )  still  denote  the  deterioration  process.  Let 
each  item  be  observed  as  it  leaves  each  processor,  and  let  tj/6  denote  the  nth 
observation  at  processor  i.  Let  0n’£  include  k*n,€,  the  ’deterioration 

measure’  between  the  n-1*4  and  nth  observed  sample  at  processor  i.  Let  S^,e 
denote  the  system  state  at  processor  i  at  the  time  of  the  nth  sample. 

n-l 

Then  =  S(«  I  kj,€).  Assume  that 

j=t 

PUi;€  €  Al^e,  k  <  n,  sft  =  s} 


f  N 

[p‘(U  +  /r  E  Bj(s)pj(5)]x(dO. 

\  i=» 


Here  under  each  fixed  system  state,  we  let  .  n  <  “}  be  i.i.d.,  and  let 


F.TVW’WWJ 


Ux,€,  n  <  “)  be  independent  of  U„'€,  n  <  •}. 

Under  assumptions  analogous  to  those  used  in  Theorem  1,  we  get  (4.1), 
where  the  H\  E‘r  K1  are  defined  in  the  obvious  way.  The  wx(  ■)  and  w2(  ) 
are  mutually  independent. 


dP.(t)  -  f  I  pu  Pj  (t)  -  E  P,i  P,(t)]dt 

L  i*  i*.  J 


(t)  i  [h,(s)  .  _ r1rdyi . ». 

i=i  l-K^Pft))  K*(P(t))  ■*  ^  Ki(P(t))J  >- 


where 


(P(t)) 


.  .  H‘(S(t))  ,  dw'(t) 

dy'  =  - dt  +  - 

K‘(P(t))  [K*(P<t))] 


and  w‘(  )  is  a  Wiener  process  with  covariance  Ejt. 

The  interpertation  as  a  filtering  equation  is  the  same  as  for  the  result 
of  Theorem  1. 

The  result  can  be  extended  to  more  general  networks,  to  the  correlated 
observation  case,  and  with  the  intermittent  sampling  extensions  of  subsection  4.1 
above. 


4.3.  Interpolation  3.  By  the  definition  of  interpolation  3,  the  limit 
processes  for  that  interpolation  can  be  obtained  from  the  result  in  Theorem  1 
by  rescaling  time  by  R(t);  in  particular,  in  interpolation  3  we  have 


where 


dP.(t)  =  K(P(t))  b,(P(t))dt 
+  P,(t)[H(s)  -  H(P(t))]'  I'1  [dy  -  H(P(t))dt], 

dy  -  H(S(R(t)))dt  +  dw,  cov  w(t)  =  E,t. 


,*  -r. A  -f.  A  «*,  A--./  AA 


nmsm 


To  understand  (4.3),  (4.4),  note  that 

Pf(t)  (in  scaling  3)  =  P*(R£(t))  (in  scaling  1). 

Equivalently,  there  is  no  K(P(t))  ’normalization*  in  the  the  observation  part 
of  the  equation  since  the  (discrete  parameter)  time  scale  is  measured  simply  by 
the  number  of  observations  taken.  The  K(P(t))  reflects  the  mean 
instantaneous  ’rate  of  deterioration’  per  sample,  given  the  observed  data.  The 
result  for  interpolation  2  is  obtained  in  a  similar  manner. 

4.4.  An  alternative  representation  for  P(-)  and  non-anticipative  stopping 
times.  To  prepare  for  the  discussion  in  Section  5,  we  use  interpolation  3,  with 
k*  =  At*  but  keep  the  other  conditions  of  Theorem  1.  Let  T£  *  0  denote 
a  tight  and  [0,“)  valued  sequence  of  random  variables,  which  together  with 

(P€(  ),  Rt(  ),S(  )}  converges  weakly  to  (T,  P(  ),  R(  ),  S(  - ))  as  t  -  0.  Let 
I(Tf<nf)  be  D*  measurable  for  each  t,n.  We  show  that  there  is  a 
N-dimensional  Wiener  process  v(  )  with  covariance  Ejt  such  that  I{T<  }> 
S(R(  •))  and  P(  )  are  non-anticipative  with  respect  to  v(  )  and 

(4.5)  dP,(t)  =  K(P(t))b,(P(t))dt 

+  P,(t)  [h(s)  -  H(P(t))]  '  Ej1  dv(t),  s  <  M. 

Of  course  v(  )  has  the  representation  dv  -  dy  -  H(P(t))dt. 

Let  g(  )  and  f(  )  be  smooth  real  valued  functions  and  let  tt  <  t  <  t 

+  T,  i  <  q.  Choose  any  t;  such  that  P(T  -  tj)  =  0.  Let  A  denote  the 

operator  of  the  system  (4.5)  (with  v(  )  a  Wiener  process  with  covariance  E,t). 
Let  Q,(P)  denote  the  row  vector  Pt[H(s)  -  H(P)]  Ej1,  and  let  Q(P)  denote 
the  matrix  with  sth  row  Q,(P). 


pnuntrewv.TtirvTV TV rs r.T .-v. v.v.’ j_v,v  ww.* nr v  *y r>- » •»  v* '■■*  a  '■s'.^.v  v.  v  ■  ■  - 
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Wc  have  (under  the  conditions  of  the  first  three  sentences  of  this 
subsection) 


(4.6) 


Eg(Pt(ti),  I{T  «  )f  S€  (R^t,)),  i  <  q) 


t+T 


[f(P€(t+r))  -  f(P4(t))  -  «  I  (f(P*+1)  -  f(P„))  /« ]  =  0 


n€  =t 


also,  the  left  side  of  (4.6)  equals  zero  if  Ej,£jtj  is  put  before  the  sum  and 
Et)£  before  the  nth  summand  (recall  that  Ej)£,t)  *  ^D€  )•  Doing  this  and 
expanding  the  Ej)£  f(P*+,)  -  f(P*)  and  using  covjj£  6V*  =  Ej£  +  o(£)  (see 


(3.9)  for  the  definition  of  6V*)  yields 


<e  -  r<p«)]  -  <r>‘)b<p;)E  it* 

n  n 

+  i-  ‘race  fpp(P<)  [Q(P<)  E,  q’(P‘)]  +  o(0. 


Here,  the  subscripts  denote  the  obvious  gradients  or  Hessians. 

Putting  this  into  (4.6),  averaging  the  sums  of  the  Ej)£  At*  as  done  in 

n 

Theorem  1,  and  letting  t  -  0  yields 

pt+T 


(4 


pti 

.7)  Eg(P(tj),  I{T<t  ),  S(R(tj)),  i  <q)  [f(P(t+T))  -  f(P(t))  -  Af(P(u))du]  =  0. 

‘  *  t 


Owing  to  the  arbitrariness  of  g(  ■),  f(  ),  {t^  and  q,  P(  )  solves  the 

martingale  problem  with  respect  to  the  oalgebras  F(  measuring  (P(u),  I {T<u}- 
S(R(u)),  u  <  t).  The  assertions  concerning  v(  )  and  the  non-anticipativcness 
property  all  follow  from  this. 

This  result  will  be  used  in  the  next  section,  where  it  will  be  necessary  to 
show  that  the  limits  of  the  stopping  (repair)  times  Tt  are  ’admissible’  stopping 
times  (i.e.,  non-anticipative  with  respect  to  the  appropriate  Wiener  process)  for 
the  limit  filtering  system. 
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5.  The  Optimal  and  Nearly  Optimal  Maintenance  Policy. 


The  optimal  stopping  and  repair  policy  for  the  physical  process  with 
observations  {(*}  is  usually  difficult  to  compute  even  in  the  simplest  cases.  It 

is  often  substantially  simpler  to  compute  a  policy  for  the  limit  process.  It  is 

natural  to  try  to  use  a  policy  which  is  optimal  or  nearly  optimal  for  the  limit 

process  on  the  actual  physical  process.  The  question  then  arises  concerning  how 

good  is  this  ’limit*  policy  when  compared  to  the  optimal  policies  for  the 
physical  processes.  We  treat  one  simple  and  classical  case  and  show  that  the 
’limit’  policy  is  indeed  ’nearly’  optimal  for  small  t.  We  choose  a  classical 
model  for  which  the  calculations  are  simple,  but  the  idea  also  works  for  the 
general  case. 

Problem  description.  The  state  space  -  I  «  (G,B)  =  (Good,  Bad),  with  Bad 
absorbing  and  pGB  *  p  >  0.  Interpolation  3  will  be  used,  let  k*  =  At*.  (*  ■ 
(l*, At*),  where  {{*,  n  <  “}  are  the  ’quality’  variables  and  are  independent  of 
(At*,  n  <  «},  and  the  At*  do  not  depend  on  the  state  or  on  t.  We  suppose 

that  (At*)  is  a  stationary  first  order  Markov  process  with  a  unique  invariant 

measure  (although  the  results  will  be  true  for  any  stationary  process  {At*} 
which  is  sufficiently  mixing.  We  now  only  need  a  model  of  the  form  (2.3)  for 
the  {£*).  We  use 

+  ^  +  O(vTF) 

(5.1)  f 

IpU.t  )Md(  )  -  0. 

Here,  we  can  always  take  N  ■  1,  since  there  are  only  two  states.  By  (5.1), 

B(G)  -  1,  B(B)  »  0,  P0U,l')  -  PB({,1').  We  use  PB  as  the  reference  measure  - 

but  it  could  be  PG  or  any  ’intermediate’  measure  -  with  the  same  results.  Wc 
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use  E°  to  denote  expectation  with  respect  to  any  of  the  stationary  measures 
either  that  for  {At*}  or  that  for  {5*}  which  corresponds  to  the  one  step 
transition  function  (see  (A2.2))  PU,l,dl')  =  PBU,t  ')X(d(  ')•  Also 


h(B,0  =  0 
h(G,0 


PJ(U') 


MdT  ) 


(5.2) 


J  P B(U  ) 

Ij  =  E°h(G,I) 

H(B)  =  0,  H(G)  =  E°h(G,I) 
H(P)  =  H(G)(1-Pb). 


=  I, 


We  assume  the  appropriate  forms  of  (A2.1)  -  (A2.5)  throughout  this 
section. 

By  the  above  definitions  and  (4.3),  the  limit  filter  is  (interpolation  3) 

(5.3)  dPB  «  (E°At)pB(l  -  PB)dt 

-  PB(1  -  PB)H(G)r,1  [dy  -  (1  -  P B)H(G)dt] , 
dy  *  dw  +  H(S(t))dt,  covar  w(l)  »  E,. 

Write  dv  =  dy  -  (1  -  PB)H(G)dt.  v(  )  is  the  innovations  process  as  discussed 
at  the  end  of  Section  4,  and  it  is  a  Wiener  process  with  covariance  Ejt.  Also 
PB(  )  is  non-anticipative  with  respect  to  v(  ).  Rewrite  (5.3)  as 

(5.4)  dPB  -  (E°At)p(  1  -  PB)dt  -  PB(1  -  PB)H(G)E'i1dv. 

If  v(  )  is  replaced  by  an  arbitrary  Wiener  process  with  covariance  Ejt,  then 

(5.4)  has  a  unique  non-anticipative  solution  PB(  ),  and  its  probability  law  is  the 
same  as  that  of  the  solution  to  (5.3)  or  (5.4).  Thus,  for  purposes  of  optimal 


control,  if  the  cost  criteria  are  in  terms  of  P0(  )  only,  and  do  not  involve 


I 


WlAAV 


S(),  then  (5.3)  and  (5.4)  are  equivalent  models.  This  is  known  as  the 
’separated’  problem  (Haussman  [1985])  in  stochastic  control  theory.  This 
replacement  or  separation  will  be  used  below. 


The  cost  criterion.  If  the  processor  is  inspected  and  found  to  be  G,  then 
we  charge  cr  If  it  is  found  to  be  B  on  inspection,  then  we  change 

c2  >  c,.  After  inspection  (and  repair,  if  B)  the  state  is  G.  Furthermore,  we 
wish  to  penalize  the  quality  loss  due  to  a  bad  machine.  Under  B  or  G,  the 
distributions  of  the  ’quality’  are  close.  In  line  with  our  view  of  ’marginal’ 
costs,  the  penalization  should  be  only  on  the  mean  difference  in  the  quality  for 
B  and  G.  In  heavy  traffic  modelling  of  queues  it  is  common  to  value  lost 

production  at  0(/r)  per  item.  With  this  in  mind,  and  using  the  0(/r) 

difference  in  the  distributions  of  quality,  a  reasonable  total  cost  over  a  time 

interval  [0,t]  is,  for  some  c0  >  0, 


E  T.  t  CQPn(B)I (complete  a  production  at  time  n} 


+  E  Y.  Cj  I  {repair  at  time  n,  G) 
n=l 
t/€ 

+  E  T.  Cj  Impair  at  time  n,  B}' 


Normalizing  and  taking  limits  yields 


(5.6)  li 


•  E  r  f‘ 

,m  ~  co 
t  1  L  Jo 


P|(u)du  +  cx[#Repairs  by  t  when  G] 


+  c2[#Repairs  by  t  when  b| 


The  observations  after  the  repair  are  assumed  to  be  independent  of  those 
before  the  repair.  We  consider  only  stationary  admissible  policies:  The  decision 


to  repair  can  depend  only  on  the  observed  data  taken  since  the  last  repair,  and 
the  decision  function  does  not  depend  on  the  number  of  past  repairs.  Let  T£ 
denote  the  length  of  the  renewal  interval  between  repairs.  Then  if  ET£  <  «°, 

(5.6)  equals 

E[/I€  copKu)du  +  ci(l  *  pi(T£))  +  caP|(Te))] 

(5.7)  =  V€(T£). 

et£ 

In  order  to  avoid  some  peripheral  details  below,  we  make  the  unrestrictive 
assumption  that  there  is  some  small  A0  €  (0,1)  such  that  repair  takes  place  on 
or  before  the  first  time  that  P|(t)  >  1  -  AQ. 

The  cost  and  optimal  policy  for  (5.4).  Define  the  cost  for  (5.4),  where 
T  is  any  stopping  time  which  is  non*anticipative  with  respect  to  v(  •). 


Theorem  2  (Shiryaev  [1978]).  Let  PB(0)  =  0.  There  is  Aj  €  (0,1)  such  that 
T  =  min  (t  :  P0(t)  =  1  -  Aj)  is  optimal  for  V(T),  with  respect  to  all 
non-anticipative  T.  Also  ET  <  ®. 

We  choose  AQ  such  that  Aj  >  AQ. 

The  limits  of  the  costs  and  optimal  stopping  times  for  P€(  ).  Let  T£ 
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denote  the  optimal  stoping  time  for  P|(). 

Theorem  3.  Let  P|(0)  =  0.  Then  (T£)  is  uniformly  integrable. 

Proof:  The  theorem  is  a  consequence  of  the  Markov  property  of  S(  )  and  the 
fact  that  P*  is  a  conditional  probability.  There  are  6L  >  0  such  that  for 
all  t  and  all  s  * 

V(t){PB(t+S)  >  1  -  Ao)  >  62  W-P-1- 

This  implies  that  all  moments  of  the  entry  time  into  the  set  [1  -  AQ,  1]  are 
bounded. 

Q.E.D. 

Theorem  4.  Let  (P|(),T£)  ^  {PB(),T},  where  T£  is  optimal  for  Pg()  and 
cost  (5.7).  Then  PB(  )  satisfies  (5.4),  T  is  non-anticipative  with  respect  to  v(  •) 
and  V€(T£)  -  V(T). 

Proof:  The  weak  convergence  is  just  Theorem  1  (see  also  Section  4.4).  The 
non-anticipativity  was  proved  in  Section  4.4.  The  convergence  Ve(T£)  -  V(T) 
is  a  consequence  of  the  weak  convergence  and  Theorem  3. 

Q.E.D. 

The  near  optimality  of  T  for  P|().  Define  the  stopping  role  T£  by 
T£  =  min  {t  :  P|(t)  >  1  -  Aj).  Theorem  5  says  that  T£  is  nearly  optimal  for 
PB(  )-  and  justifies  using  the  limit  model  (3.1)  or  (5.4)  for  control  purposes. 


Theorem  5. 


lim  [V€(T£)  -  V€(T£)1  >  0. 

€ 

Proof:  Select  a  weakly  convergent  subsequence  of  {1 
indexed  by  €.  Then  (P|(),  T£}  =*>  (PB(),T)  and 

A 

where  T  is  non-anticipative  with  respect  to  v() 
and  the  weak  convergence  V€(T£)  -»  V(T),  V€(T£)  -* 
optimal,  V(T)  €  V(T). 


P£(-),T£),  {P|(  ),T£},  also 
{P£()>T£}  =>  {Pb(  ),T}, 
in  (5.4).  By  Theorem  4 
V(T).  But,  since  T  is 
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